Efficient Data Representations for Information Retrieval

The key role compression plays in efficient information retrieval systems has been recognized for some time. However, applying a traditional compression algorithm to the contents of an information retrieval system is often not the best solution. For example, it is inefficient to perform search operations in maximally compressed data or to find the intersection of maximally compressed sets. In order to perform these operations, the data representation must be fully decompressed. This thesis explores practical space versus time trade-offs which balance storage space against the competing requirement that operations be performed quickly. In particular, we are interested in variable length coding methods which are both practical and allow codeword boundaries to be found directly in the compressed representation. The latter property allows considerable flexibility in developing algorithms which can manipulate compact sets and sequences and allow selective decompression. Applications of such coding methods are plentiful. For instance, variations of this theme provide practical solutions to the compressed pattern matching problem. They are also a vital element in many of the compact dictionary representations recently proposed. In this work, we propose new data representations which allow key query operations to be performed directly on compressed data. This thesis draws together previous work and shows the fundamental importance of coding methods in which control information is built directly into the representation. More particularly, this thesis (a) reviews current applications of compression in string searching algorithms; (b) critically evaluates existing coding approaches which allow fast codeword length identification; (c) introduces a new coding method which shares this property; (d) investigates applications of these coding approaches to searching and seeking in compact sequences; and (e) explores new data representations which provide a compromise between good compression and fast querying in compact sets.